Augmented reality for assisting consumer transactions

ABSTRACT

A portable device which assists shoppers. The device identifies items of merchandise, and retrieves information about the items from a remote location. The items can be identified based on their labels, by image recognition, by their location, by the shopper, or by other approaches. This additional or enhanced information about the items can then be overlaid on the real-time display of the portable device, such that the shopper has access to said information and can associate it easily to the relevant items.

The invention concerns portable devices which obtain information aboutitems within a retail store, or other venue. The information augmentsthat which is already known to the customer.

BACKGROUND OF THE INVENTION

The widespread use of computers has made available a vast amount ofinformation to consumers. When a consumer embarks on an ordinaryshopping trip, or a special trip to make a large purchase, the consumercan make more intelligent choices if he or she is assisted by relevantinformation. However, such information can be cumbersome to obtain in asimple manner, and the consumer often does not know in advance whatinformation they need or which products they will find interesting atthe store.

The invention facilitates intelligent shopping by making usefulinformation readily available to consumers within the store premise.

OBJECTS OF THE INVENTION

An object of the invention is to provide information about items which aconsumer considers for purchase, in a simple manner.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a situation in which the invention 7 can be used.

FIG. 2 is a schematic view of device 7 of FIG. 1.

FIG. 3 is a flow chart of processes undertaken by one form of theinvention.

FIG. 4 is a bird's eye view of the interior of a venue 3 of FIG. 1,taking the form of a retail store containing shelves 50.

FIG. 5 is a variant of FIG. 4.

FIG. 6 illustrates a landscape 100, and represents a generic view takenwithin the venue 3 of FIG. 1

FIG. 7 illustrates the image of FIG. 6, together with augmentation addedby the invention.

FIG. 8 is a block diagram of one form of the invention.

SUMMARY OF THE INVENTION

In one form of the invention, a person carries a computing device into aretail store. The person uses the device to (1) identify nearbymerchandise, and then (2) retrieve information about the merchandise,from a local network, the Internet, or both.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows a geographic site or venue 3, which a person 6 is about toenter. The venue 3 may take the form of a retail store, retail mall,museum, national park, sports stadium, zoological park, theater,exhibition hall, military cemetery, ancient battlefield, or any locationwhich contains features or objects about which the person 6 would liketo obtain information.

The person 6 is equipped with a portable computing device 7, whichreceives the information described below, and displays it. Device 7 cantake the form of a mobile phone (or “smart phone”) which has an onboardcamera and access to wireless and/or cellular data networks. Other formsof the computing device 7 might be a portable computer or a mobiletablet, appropriately programmed, and equipped with wireless access toexternal networks.

In FIG. 1, block 9 represents a computing facility which is under thecontrol of the agency which owns, or manages, the venue 3. Block 9 canmake information available to the person 6 by way of the Internet 12,through wireless linkages. Alternately, a local network, such as WiFi,can be used, which is under the direct control of the venue 3, and notinvolved in public transmission of data, as is the Internet.

Block 15 represents another computing facility, but under the control ofa third party, which can make additional information available to theperson 6. Block 15 can include a computing facility, and can communicateover the Internet 12. The additional information can be developedindependently of the agency which operates the venue 3, and acts as asecondary source of information for the person 6.

In one form of the invention, block 15 communicates with device 7exclusively through a public carrier, such as the Internet 12, and block9 communicates exclusively through a non-public medium under control ofthe venue 3, such as a local wireless network 10.

Some generalized examples are given which illustrate operation of theapparatus of FIG. 1, followed by a detailed discussion of techniques ofimplementation.

FIRST EXAMPLE

A first example relates to purchasing wedding gifts. Venue 3 is a retailstore which sells wedding gifts, and other items. In the United States,it is common for couples who are about to be married to list themselveson a registry, which indicates their preferences for wedding gifts.

Under the invention, the registry is retrieved by block 9 from thestore's server, and transmitted to the device 7. This allows the person6 to ascertain which gifts on the registry have been purchased, andwhich remain unpurchased, so that duplicate gifts are not purchased forthe couple.

SECOND EXAMPLE

A second example is perhaps most applicable to the larger retail stores.Block 15 stores a map of the store, which indicates locations of all thegoods held for sale. The device 7 downloads the map from block 9. Thismap assists the person 6 in locating items desired for purchase. Anindex, or legend, indicates the locations of items on the map, tofacilitate location of specific items.

THIRD EXAMPLE

A third example is a type of enhancement of the second example. Theperson 6 prepares a shopping list, loads the list into device 7, andvisits the venue 3. Device 7 cooperates with block 9 to guide the person6 through the venue 3 in the most efficient manner, to obtain the itemson the list.

Known software routines in the network sciences, such as those used tosolve the “traveling salesman” problem, can be used to establish therouting.

FOURTH EXAMPLE

In a fourth example, the person 6 utilizes device 7 to obtaininformation about a specific product encountered by the person 6 at thevenue 3. The device 7 identifies the product, such as a coffee grinder,and then obtains information about the product through block 9. Theinformation can include data on price, warranty, number of items held instock, sizes and colors available, and so on.

Multiple levels of information are possible. A first level is determinedbased on statistical sampling of customers, and includes the informationrequested by most customers, and is presented by default. A second levelcan include more detailed information, such as installation instructionsor material hazard data, and so on. The information of the second andfurther levels is presented only if requested.

FIFTH EXAMPLE

In a fifth example, block 9 transmits additional information to device7, such as descriptions of goods which are frequently purchased alongwith the coffee grinder. Alternately, block 9 can transmit informationof general interest.

This additional information can promote goods which are complementary tothe item in question. Alternately, the additional information candescribe similar goods of different quality, or having differentfeatures, or which are considered substitutes for the coffee grinder.

It is pointed out that device 7 itself receives and displays theinformation, and not a device under control of the venue 3, such as aself-service kiosk provided by the venue 3. This provides severaladvantages. One is that the person 6 is familiar with the operation ofhis own device 6, and need not learn how to operate a strange machineprovided by the venue 3.

A second advantage is that it is common for people to encounter publicinformation kiosks which are malfunctioning. It can be difficult to tellwhether a delay in responding by such a kiosk is the result of amalfunction, or is a normal delay. Use of device 7 eliminates the issueentirely.

A third advantage is psychological. Many people will place trust ininformation retrieved by their own devices 7, as opposed to informationdisplayed on a possibly malfunctioning public kiosk. Restated, itappears that when a person becomes familiar with the normal sequence ofevents of the person's own device 7, the person attributes trust to thedata when that normal sequence occurs. However, a public kiosk willprobably undertake a different sequence so that, even if the sameinformation is presented, doubt seems to arise in the person's mindbecause of the unfamiliar sequence in presentation.

It seems axiomatic that the familiar is trustworthy. The invention seeksto make that which is familiar available to the customer.

SIXTH EXAMPLE

In a sixth example, the venue 3 is a zoological park. The information ofthe type in the fourth example, above, would relate to the animals wholive there, and their habits. The information of the fifth example couldrelate to books about the specific animals, which are available forpurchase.

Similar principles would apply if the venue 3 is an automobile raceway,where the information provided concerns the race cars, or drivers. Ifthe venue 3 is a baseball park, the information provided would concernthe players, their team, and so on.

The processes of obtaining information, described above, apply to thevarious types of venue enumerated above.

SEVENTH EXAMPLE

In a seventh example, the venue 3 is an automobile dealership. Block 9can make information available about the cars, which information isprovided by the dealership, or the manufacturer. Block 15 can makeavailable independent evaluations made by a testing service, such as theConsumers Union in the United States.

EIGHTH EXAMPLE

In an eighth example, the venue is again a large retail store. Theperson 6, in general, cannot see from a single location the signs whichdescribe the contents of all shelves in the store, and thus, forexample, cannot immediately locate garden hoses. Block 9 transmits a mapto device 7, together with an index. The person 6 locates garden hosesin the index, and then device 7, perhaps together with block 9, indicatethe location of garden hoses on the map, which is displayed on device 7.

NINTH EXAMPLE

In a ninth example, the location of the device 7 within the venue 3 isascertained. That location also indicates, by inference, the location ofthe person 6. Block 9 can then use that location information to furtherassist the person 6.

For instance, the person 6, through device 7, can request assistance ofa staff member of the venue 3. Since block 9 knows the location of thedevice 7, block 9 can direct the staff member to that location, which isthe location of person 6.

Alternately, knowledge of the location of the person 6 can allow block 9to direct the person 6 to a specific product, by giving the person 6directions, without direct, in-person assistance of a staff member,although a staff member may be involved in sending messages whichcontain the directions.

TENTH EXAMPLE

In a tenth example, a second device 7 is used, and attached to a smallchild who belongs to person 6. This second device 7 may include a serialnumber which block 9 can interrogate. If person 6 develops the beliefthat the child has wandered away, person 6 can request, through device 7or otherwise, block 9 to locate the child, by way of locating a device 7having the serial number. Device 7 could be linked to an in-storetracking system, such that anyone attempting to remove the child fromthe store through any of the monitored doorways would raise an alarm,both to store personnel and to the parents' (e.g. person 6) mobiledevices.

ELEVENTH EXAMPLE

Many large retail stores also provide (1) automotive services, such asthe sale of tires and other parts, and (2) automotive repairs. Person 6may leave a car to be serviced, and, when service has finished, venue 3can notify person 6 of that fact, by transmitting a message from block 9to device 7.

In addition, if extensive servicing is expected, venue 3 can notifydevice 7 of the progress being made, together with an estimated time ofcompletion.

TWELFTH EXAMPLE

The device 7 can act as an advanced loyalty card, with benefits providedto the person 6. The benefits can include discounts, or standard loyaltypoints.

In one form of the invention, the location of device 7 within the venue3 can be ascertained. That will allow the managing agency of the venue 3to track the movements of person 6, and to learn, for example, where theperson 6 spends most time, and what places the person 6 avoids. It ispointed out that a standard loyalty card provides different information,namely, information as to what a customer actually purchased, not wherein the venue 3 the customer spent time.

In addition, the requests for information about products made by thedevice 7 indicate the interests of the person 6.

Tracking the activities of device 7 and consumer 6 in “real-time” canprovide useful market research.

THIRTEENTH EXAMPLE

The invention provides a virtual dressing room for purchasers ofclothing and other items of apparel. The device 7 is equipped with avirtual mannequin which matches the physique of person 6. That is, thevirtual mannequin is a three-dimensional CAD (Computer Aided Design)data set, or wire-frame model, which corresponds in shape to person 6.This virtual mannequin can be developed, for example, by measurementstaken by a professional tailor.

The person 6 then identifies an article of clothing at the venue 3,using device 7. Block 9 contains a collection of virtual costumes, whichincludes the article selected. The device 7 then places the selectedcostume onto the virtual mannequin, and displays the results. Since thedisplayed results are based on three-dimensional CAD data, the mannequincan be rotated, bent, and so on, to illustrate the appearance of thecostume in different poses.

The virtual mannequin can be equipped with an actual photograph of thehead of person 6, which is scaled to match the mannequin. In addition,several photographs of the head can be provided, taken from differentangles, to match different poses in which the mannequin can be placed.

A default set of poses, which correspond artistically to the views ofthe head in the several photographs, can be offered, to the exclusion ofother poses. The latter poses may look awkward, or even be impossiblephysiologically, because the position of the head, in a photograph, maynot match visually a selected pose.

To reduce the latter problem, an actual CAD rendition of the person'shead can be generated, so that a near-infinite number of positions canbe selected, to match poses of the mannequin.

Still further, renditions of the costume, and/or the head, can be madein different lighting conditions, such as high-noon daylight, candlelight, fluorescent factory lighting, and so on. These lightingconditions can provide a more accurate prediction of how the costume orapparel will look in the environment where the costume is expected to beworn.

Posable virtual mannequins, with selected costumes, are commerciallyavailable, as through the software package known as “Poser,” availablefrom Curious labs, Santa Cruz, Calif., USA.

FOURTEENTH EXAMPLE

This is a variant of the thirteenth example.

People who wear eyeglasses purchase eyeglass frames. Optical shopscontain numerous sample frames.

But when a person tries to select a sample frame which suits his/herface, the sample frame will not contain proper lenses which match theperson's eyes. Consequently, the person cannot accurately see in amirror how the frame looks on his/her face. This is particularly true ifthe person has just had an eye examination, and, as a result, theperson's pupils are dilated.

Under one form of the invention, device 7 is equipped with a photographof the face of person 6. The person 6 selects a sample frame at anoptical shop. Block 9 delivers an image of the selected frame, which isadded to the photograph of the face. The image of face-plus-frame isdisplayed by device 7. In this manner, the person 6 can look at thecombined image, displayed on device 7, through his/her own correcteyeglasses, and accurately see the image. This approach is in contrastto looking in a mirror without wearing eyeglasses, in the attempt toexamine sample glasses frames, lacking lenses, which are perched on thenose.

In a simple implementation of this approach, a straight-on head shot ofthe person 6 is obtained. An analogous shot of a sample frame isobtained. Both images are bitmaps, as in the JPEG format. The bitmap ofthe frame is scaled by computer adjustment to match the head shot, ifnecessary, and pasted onto the head shot.

In one approach, a real-time photograph, or video image, of the person 6is taken by an optical shop. That image is displayed on a screenmaintained by the shop, or perhaps on device 7. Then an image of theselected frame is overlaid onto that image of the face.

Technical Aspects

It was stated above that the device 7 in FIG. 1 identifies an itemwithin the venue 3, such as a coffee grinder on display at a retailstore. This identification is then transmitted to block 9, to obtainfurther information about the item. Several approaches to making theidentification will be discussed.

It is pointed out that “identification” of a coffee grinder does notnecessarily mean simply identifying it as within the generic class ofcoffee grinders. It can also mean identifying the coffee grinder withsufficient specificity, by manufacturer and model number, so that block9 knows which of the many coffee grinders stocked by the store is beingreferenced. This latter specific identification allows block 9 toretrieve information relevant to that particular coffee grinder, to theexclusion of other coffee grinders.

Several approaches are available to identify the items.

APPROACH 1. This approach uses image recognition, and alsooperator-assisted image recognition.

The device 7 in FIG. 2 is equipped with a camera 30 which captures animage of an object, and produces a digitized image of the object. Imagerecognition software 33, known in the art, analyzes the image andidentifies the object.

Another approach is to capture an analog image, take a Fouriertransform, and then identify the object by comparing the Fouriertransform with Fourier transforms of reference objects. In a variation,a digital image can be taken of the object, and a Fourier transform ofthe digital image can be used, as just described.

Under either approach, it may be helpful for the human operator toassist the image recognition software. For example, the operator mayinitially inform the software that the object to be identified iscontained within a particular class, such as golf equipment. This willprevent the software from mis-identifying a golf putter as a hockeystick.

Alternately, this assistance may be deduced from the location of thedevice 7. Determining location is discussed later. For example, if thedevice 7 is deduced to be near golf equipment, then it is assumed thatany image taken by the device 7 will be of golf equipment, and not ofcleaning supplies.

It may not be necessary that the recognition software conclusivelyidentify an object, but it may be sufficient if the software concludesthat the object in question is one of a fixed number of possibilities.For example, the software may reduce the possibilities to a 5×5 matrix,containing 25 possible objects in total, which can be displayed on asingle computer screen. The person 6 then selects the correct object.

Further, this latter approach may be continuously interactive with theperson 6, while the software deduces the identity of the object inquestion. For example, the software may initially deduce the size of theobject, and may ask for confirmation that the deduction is correct.

In the case of size determination, the software may not ask the person 6to indicate an absolute size, in inches, but may be equipped withseveral common reference markers as to size, such as (1) wristwatch, (2)peanut butter jar, (3) cigar box, (4) loaf of bread, (5) suitcase, (6)steamer trunk, (7) easy chair, and so on. Initially, when the softwarehas reached a preliminary conclusion as to size, it will ask a questionsuch as “Is the object about the size of a loaf of bread?” and theperson will answer appropriately.

The software will then advance to other categories, such as weight,cost, type of use, and so on, and ask for confirmation of itspreliminary conclusions.

Therefore, in one approach, the device 7 identifies an object ofinterest within the venue 3 through image recognition. In one variation,the person 6 assists the identification, by correcting the conclusions,or providing information as to characteristics of the object.

In the latter case, the image recognition software contains variouscategories, such as size, weight, color, and so on. The software asksthe person 6 for specific values in each category, such as the specificsize, specific weight, specific color, and so on, to assist itsdeduction of the identity of the object of interest.

In another variation, the software does not reach a conclusion as to theunique identity of the object in question, but instead reduces theuniverse of possibilities to a specific number of identities, such asthe number 25 stated above. Then the person 6 selects the correctidentity from that reduced universe.

For example, the object in question may be a sports car, with aconvertible top. The software would respond by listing thepossibilities, and the person 6 would select the correct make and modelfrom the possibilities presented.

APPROACH 2. Quick Response codes, QR codes, are known in the art. A QRcode allows a person X to store data on the Internet, and a person Y toscan the QR code, and obtain access to the information.

QR codes, at present, resemble small checkerboards, roughly the size ofpostage stamps, each having a specific pattern of squares being coloredin a specific way.

For example, assume a checkerboard having 10×10 squares, or 100 squarestotal. Assume that each square is either black or white. Thischeckerboard is, in essence, a 100-bit binary number, and can uniquelyidentify a number equal to (2 raised to the 100th power) different items(minus any bits reserved for error detection and correction).

Of course, larger numbers of squares, and larger numbers of colors, canincrease the possible binary numbers which a QR code represents.

QR codes can be attached to the objects, to identify them. Or QR codescan be attached to known points in the venue 3 of FIG. 1, to allowdevice 7 to determine its location. Determination of location isdescribed in greater detail below.

QR codes are commonly used as follows. A QR provider creates a multitudeof QR codes, and sells them. The QR provider also creates a web site, orequivalent, and assigns each QR code to a space within the web site. TheQR provider also maintains an index, which contains an entry for each QRcode, that is, each number represented by each QR code has a line in theindex. Each line points to the space at the web site assigned to the QRcode.

For example, assume that Buyer A purchases QR code A, which is assigneda space A. Buyer A places information into the space A. When a thirdparty optically scans the QR code A, that third party then consults theindex, and thereby learns the address of the space A within the QRprovider's web site. The third party can visit that space A, andretrieve information which buyer A has placed there.

Therefore, a QR code allows a person X to store data on the Internet,and a person Y to scan the QR code, and obtain access to theinformation.

A QR code can be assigned to each object described herein. The datadescribed herein, which is associated with the item, is stored in thespace assigned to the QR code. This data would be stored in server 9 inFIG. 1, although other locations are possible. A person who scans agiven QR code obtains access to the stored data.

In one form of the invention, the operator of the venue 3 in FIG. 1 canhandle storage of the data corresponding to the QR codes.

APPROACH 3. The device scans the UPC, Uniform Product Code, of an itemto identify it. Alternately, the user of the device types the UPC intothe device, analogous to the situation when a check-out clerk cannot usea scanner to read an illegible UPC.

APPROACH 4. Under this approach, device 7 identifies items based on theposition of device 7 within the venue 3. How device 7 determines itsposition is explained later. But once the device 7 deduces its position,it can learn the identities of the item or items which are near it, froma map provided by the manager of the venue 3.

Under this approach, the manager of the venue 3 maintains data whichindicates the spatial distribution of the items within the venue 3: themanager maintains a map showing where all items are located. The data,or map, is made available to device 7 through server 9.

Device 7 then (1) ascertains its current position within the venue 3, asin block 40 in FIG. 3, and (2) retrieves data which describes the itemslocated at that position, within a convenient distance, such as tenfeet, as in block 43.

Preferably, the convenient distance is chosen so that the number ofitems listed will fit onto a single display screen of the device 7. Forexample, if the items are automobiles at a dealership, the convenientdistance will be large. If the items are diamond rings, the convenientdistance will be small.

The person 6 then selects an item of interest from those nearby listedon the device 7, as in block 46, and device 7 retrieves and displaysinformation about the item selected. Of course, if a single item ispresent nearby, no selection operation is required.

The transfer of the data, or map, to the device 7 can be accomplished bya network which is accessible to customers, such as a WiFi network. Item9 in FIG. 1 can represent such a network.

The device 7 provides the person 6 with the option of determining theconvenient distance discussed above. If the person 6 selects asufficiently large convenient distance, then the items displayed, or themap containing them, will include the entire venue 3 of FIG. 1. Thedevice 7 provides a zoom-in, zoom-out function to navigate through themap.

In one embodiment, the device 7 displays the map, with items in theirappropriate positions. The person 6 positions a cursor on the map whichindicates the person's position. In response, the device locates allitems located within the convenient distance from the cursor, anddisplays them. The user selects an item. In this embodiment, the person6 is the agency which inputs data indicating the position of the device7 to that device 7. The device 7 does not infer its own position, nordoes block 9.

In another form of the invention, the position of the device 7 withinthe store is determined without human intervention. Robotic systems areknown for determining the position of objects, such as automatedtrolleys, within a building.

APPROACH 5. The person 6 can undertake an interactive description whichprogressively refines a definition of an object sought, until an exactdescription is attained. Thus is done using the device 7, in conjunctionwith communication with server 9 in FIG. 1.

For example, assume that the person 6 is looking for an air filter for avehicle. The device 7 can first ask whether the vehicle is a passengercar or truck. Then the device 7 asks for the manufacturer, also called“make,” as in “maker,” (eg, General Motors, Chrysler, etc.). Then thedevice 7 asks for the year of manufacture. Then the device asks for themodel (eg, if General Motors is the manufacturer, some possible modelswould be Chevrolet, Cadillac, and so on).

This process will progressively limit the number of possible air filterswhich will qualify, until the questions finally lead to the identity ofthe vehicle in question, which, in effect, specifies the required airfilter. Then device 7 indicates where in the venue 3 the air filter islocated, by consulting data provided by the manager of the venue 3.

This process is different from a similar process in which the personidentifies an air filter using a self-service kiosk at a store. Thisprocess has the psychological benefits discussed above.

APPROACH 6. The device 7 is equipped with a GPS, Global PositioningSystem, which indicates the latitude and longitude of the device. Block36 in FIG. 2 indicates the GPS. Knowledge of those coordinates allowsdeduction of the building, or business, at which the device is located,by reference to known tables. A GPS is also capable of indicatingposition within the venue 3, with an accuracy of at least a few feet.

Alternately, the venue 3 may be equipped with RFID (Radio FrequencyIDentification) tags, which are distributed throughout the venue 3. Forexample, if the venue is 100×100 feet in size, then 100 squares, each10×10 feet, can be defined in the venue 3, like a checkerboard. Each ofthe 100 squares can be given a number.

Each square is equipped with an RFID tag. Device 7 is equipped with anRFID tag reader. The person 6 orders the device 7 to interrogate anearby tag, and thereby the device 7 learns the 10×10 square in which itis located.

FIG. 4 illustrates a venue 3, which contains ranks of shelves 50. Blocks53 indicate RFID tags which are distributed throughout the venue 3. Eachtag 53 contains data, such as a number, which identifies its location.

In another embodiment, each square in the 10×10 example above can belabeled with a visual label, which contains a number, or othercoordinates. The person 6 examines the labels, and manually enters thelabels, or coordinates, into device 7, which then ascertains itsposition on the map discussed above.

APPROACH 7. The device 7 in FIG. 5 can be equipped with a rangefinder60, as used by golfers or the military. The rangefinder 60 allowscomputation of distance D to a target item T. From this distance D, andthe compass direction along which D runs, which is deduced using a knowncompass 63 within the device 7, latitude D1 and longitude D2 in FIG. 5can be deduced, or equivalent x-y coordinates can be deduced.

Thus, the latitude and longitude of the target item T becomes known,which indicate the location within the venue 3, and the identity of thetarget item T can be inferred from known tables. The tables, or maps,list all items according to their locations. The tables are provided,for example, by the managing agency of the venue 3.

This approach allows the person 6 in FIG. 1 to identify an item T inFIG. 5 which is located at a specified distance D, in a specifiedcompass direction, with respect to the device 7. In effect, the device 7identifies the polar coordinates of the item, with respect to itself.Device 7 acts as the origin. Then device 7 converts those relative polarcoordinates to the coordinates in which the item T is listed on thetables, or map.

APPROACH 8. The venue 3 makes available tables which indicate theidentities of items located at stated positions within the store. Theperson 6 using the device 7 identifies an item using the tables. Table1, below, provides an example.

X-coordinate Y-coordinate Item present X1 Y1 Item A X1 Y2 Item B X2 Y1Item C X2 Y2 Item D

The device 7, or the person 6, or both cooperatively, locate the itemwithin the tables, and the tables provide the location, relative to themap, of the item. In this manner, the location of the device 7, incoordinates which block 9 can understand, is obtained. But the locationof device 7 is deduced from the identity of nearby items.

In a simpler version, a nearby class of products, such as women'sshampoos, is identified, and the class identified is communicated toblock 9. The fact that device 7 is located near women's shampoosgenerally may provide sufficient accuracy in some situations.

If the item used for location identification is located on a shelf, andif other items are above and below it, so that multiple items possessthe same coordinates, the device 7 reports those multiple items, andallows the person 6 to select the desired item from the collection ofitems which share those coordinates.

The tables under consideration are made available to the customer in aknown manner, as by being posted at a web site, or downloadable bydirect transfer to the device from a machine at the store.

Interim Conclusion

Therefore, as so far described, the device 7, perhaps in cooperationwith person 6, identifies an item within the venue 3. The computersystem of block 9 then transmits information about the identified itemto the device 7.

The identification of the item can be done by including the position ofdevice 7 within the venue 3, to narrow the universe of possible items.For example, as explained above, the venue 3 can provide a map on whichitems for sale are identified. If the location of the device 7 is known,then the items which are located within a fixed distance become known.

The identity of the item, or group of items, may be deduced based on thelocation of the device 7 which is near the items. For example, if thelocation of device 7 is known, then maps or tables can indicate whichitems are present within a radius of, for example, ten feet. Those itemsare presented to the person 6 in FIG. 1, by way of the device 7, and theperson selects an item of interest. The item selected is thusunambiguously identified to the venue 3, because the venue 3 initiallypresented the item to the person 6, within the group of items.

Conversely, the location of the device 7 can be deduced by the reverseprocess. The person 6, or the device 7, individually or cooperatively,identify nearby items. Then, based on maps or tables, the location ofthe device 7 is deduced, based on the identities of the items.

The preceding discussion has been framed largely in terms of obtaininginformation about a subject which is viewed by the portable device 7.Another aspect of the invention concerns the manner of displaying theinformation, once obtained.

FIG. 6 illustrates a landscape 100, and represents a generic view takenwithin the venue 3 of FIG. 1, such as a view of (1) shelves in a retailstore, (2) exhibits in a museum, and so on. In FIG. 6, the portablecomputing device 7 is shown, which is present at the landscape, anddisplays an image of part of the landscape 100.

FIG. 7 illustrates that image, together with augmentation in the form ofseveral FACTs. The image of FIG. 7 can be described as an augmentedreality, in the sense that the image of the landscape 100 of FIG. 6 actsas the reality, and the FACTs of FIG. 7 represent augmentation of thatreality.

For example, in the retail scenarios described above, the image of FIG.7 would be replaced by images taken of the interior of a retail store.The FACTs of FIG. 7 would be replaced by information which is ofinterest to the customer in possession of the device 7. Such informationcan be included within banners on the device 7 in FIG. 7, and canindicate items which are currently promoted by the store. The bannerswould point to locations on the image which correspond to the actuallocations in the store where the promoted items are located. In thismanner, the attention of the customer is drawn to the actual locationwithin the store where the promoted items reside. In another embodiment,the FACTS of FIG. 7 might “hover” near an associated element in thescene. Using the aforementioned store shelf example, a banner containingthe price of an item and the number of units in stock might hoverimmediately above a particular merchandise item.

As another example, in the shopping list scenario described above, theimage of FIG. 7 would be replaced by an image seen by the device 7within the retail store. The invention ascertains whether itemscontained on the customer's shopping list are present within that imageand, if so, one or more FACTs of FIG. 7 would indicate (1) the identityof the item found and (2) the location of that item within the image.The indicated location tells the customer where in the store the item islocated. Additional FACTs could present information about the item, asdescribed above.

Therefore, in a general sense, the device 7 in FIG. 7 captures an imageof a view seen by the person operating the device 7. That view is thereality under consideration. That reality is augmented by informationwhich is superimposed, or overlaid, onto the image. The information canbe static, as in print information, or can be dynamic, as in a video orcartoon. The augmentation information can also be dynamic in the sensethat, if the customer takes a sequence of different images as thecustomer moves through the store, or venue 3 in FIG. 1, then theaugmentation will continually change accordingly.

The augmenting information can be of varying degrees of opacity, so thatit need not obscure the underlying view. As a specific example, theaugmenting information can take the form of hollow text which crawlsacross the display, in the manner of movie titles. The underlyingreality can be seen through the hollow parts of the text, as well asthrough the outlines of the text, if the outlines are not completelyopaque.

In some of the scenarios described above, the invention offered a typeof guidance to the person 6 in FIG. 1, as in guiding the person to alocation within a retail store.

Other types of guidance are possible. For example, if the person 6 inFIG. 1 is a technician repairing an aircraft component, the device 7 cantake a picture of the component. The component can then be identified asdescribed above, as by image recognition, or assisted recognition. Thepicture of the component becomes the underlying reality.

Then the device 7 can offer guidance to the technician. For example, ifthe technician wishes to open the component, a certain latch may requireactivation. The device 7 can present a photograph of the latch, togetherwith a banner pointing to a location on the image where the latch islocated. Thus, two prompts are offered to the technician: (1) what thelatch looks like and (2) where it is located.

In FIG. 8, the photograph can be represented by block 215, and the latchrepresented by circle 205, to which an arrow points.

The device 7 can repeat this process, to assist the technician inreplacing components. The technician is given an image of a component tobe replaced, together with augmentation which indicates where thecomponent is located within the reality of the initial image.

In addition, the device can explain to the technician how to connecttest equipment to a sub-component. For example, the sub-component isfirst located within the component, as described above. The device 7instructs the technician to take an image of the sub-component. Thedevice 7 then augments the image with a banner which shows testequipment, such as an oscilloscope, together with images of the testleads in this case. Additional augmentation banners show where toconnect the leads (illustrated in an augmented banner) to thesub-component (shown in an image taken by the technician).

Therefore, the invention augments a reality perceived by the person 6 inFIG. 1. That reality can take the form of a digital photograph taken bycomputing device 7. That device 7, perhaps in conjunction with otherapparatus, deduces information about the reality which would be ofinterest to the person 6.

That deduction process can involve current active input from the person6, as when the person submits a shopping list. Alternately, thatdeduction process can involve passive input from the person 6, as when(1) person 6 owns a loyalty card of a retail establishment, (2) person 6is currently visiting that retail establishment, and (3) historical dataindicated by the loyalty card is consulted. The historical datarepresents passive input.

Once deductions are made which identify information which would be ofinterest to the person 6, such information is displayed on device 7, byoverlaying the information on the image of the reality to which theinformation relates, as in FIG. 7.

Then the process can repeat, as the reality changes, as would occur asthe person 6 moves through the venue 3 of FIG. 1.

The Inventor points out that, in the case of person 6 traveling througha retail store, the underlying reality will change continually. Thereason is that (1) the person's location in the store will change, (2)the image of the reality viewed by the person will change, or (3) acombination of (1) and (2). (As to item (3) it is pointed out that, ifthe person stands in one place, but takes an image facing north, andthen an image facing east, the underlying reality will change, yet, in asense, the person has not moved.)

As the underlying reality changes, the invention augments it withchanging information, superimposed on the image of that reality.Further, in the retail example, the choice of information can depend oncharacteristics of the person, such as past purchasing behavior, ordemographic data obtained from a loyalty card.

In one approach to providing the augmentation information, the retailstore will initially identify a multiplicity of possible underlyingrealities, such as the numerous views of the shelving within the storewhich can be taken by the device 7. The operator of the storedetermines, for each underlying reality, a universe of information. Theoperator then selects a proper subset of that universe according toinformation supplied by device 7, and transmits the subset to thedevice.

For example, if the underlying reality is a view of shelves containingbath soaps, the universe will be of information relating to such soaps,such as chemical content, health effects, prices, and so on. Then, whenone of the realities is present on the display of the device 7, theinvention selects information from the universe which is relevant tothat reality. The selection can be made partly based on characteristicsof the individual.

FIG. 8 illustrates a generalized view of one form of the invention. Thedevice 7 displays an image 200, which represents a reality currentlyvisible to the device 7. The image 200 contains an item 205 which isconsidered to be of interest to the person 6 of FIG. 1.

Blocks 215, 220, 225, and 230 represent a collection of sets ofinformation which are relevant to the item 205, and the arrow pointingto item 205 is displayed to indicate that relevance. That is, the arrowextending from item 215 indicates that item 215 is more relevant to item205 than to item 210.

For example, image 200 may represent an automatic transmission in amotor vehicle. Item 205 may represent an inspection port, covered by adoor. Block 215 may represent instructions on how to open the door.

Block 220 may represent a view of components which are located behindthe door. Block 225 may represent a test procedure to be performed on acomponent within that view. The blocks of information need not bedisplayed simultaneously, if space does not permit, but can be cycledthrough by the user.

Similar comments apply to item 210, and blocks 235, 240, 245, and 250.

The informational blocks such as 215 can be viewed as footnotes to anitem 205, and can have the normal characteristics of footnotes. Forexample, a footnote generally provides detailed information which ishighly specific, and not needed in a general discussion.

Further, block 220 can be a footnote to block 215, in the sense thatblock 220 may have no clear relevance to the underlying reality 200, butis relevant to block 215. For example, the underlying reality 200 may bea culinary dish, which may contain onions, but the onions are notapparent from the image of the reality 200.

Block 215 may list the ingredients of the dish, including the onions.Block 220 may be an explanation of how to cut and sauté the onions in aparticular manner. Thus, block 220 would be relevant to block 215, butnot obviously relevant to image 200.

A generalized procedure undertaken by the invention is the following.The person 6 in FIG. 1 uses the device 7 to take a picture of acomponent 200 in FIG. 8. The component 200 may be located in the venue 3in FIG. 1, and is part of an external reality which is visible to theperson 6. Multiple components 200 may be present, and may appear on thedisplay of the device 7.

Device 7 is a general purpose device which accomplishes functions ofinterest to a generalized person 6 in FIG. 1, such as a digital camera,or portable computer. Device 7 is not, for example, a scanner used by anemployee of a store to read bar codes and transmit information to aninventory control system of the store's computer.

In one embodiment, the component 200 in FIG. 8 is not under the controlof the person 6, as when the component 200 is an item for sale in aretail store, or is an exhibit in a museum. In another embodiment, thecomponent 200 is under the control of the person 6, as when component200 is a transmission being serviced by person 6.

The device 7 transmits, to an agency at a remote location, informationwhich allows the agency to identify the component 200. In oneembodiment, the agency is not under control of the person 6. In anotherembodiment, the agency can be an affiliate, such as an employer ormilitary superior, of the person 6.

In one embodiment, the identifying information is insufficient for theperson 6 to identify the component 200. For example, a bar code does notallow person 6 to identify the component 200. Similarly, if geometriccoordinates of an object are deduced, as described above in connectionwith FIG. 5, those coordinates do not allow the person to identify thecomponent 200. As a specific example, in one embodiment, themanufacturer or model number, or both, of component 200 would identifythe component 200 to the person 6, and that data is not transmitted tothe agency.

The agency identifies the component 200, locates information relating tothe component 200, and transmits the information to the device 7. Thedevice 7 places the information onto its display, overlaying theinformation onto the image of the component 200, in the manner of blocks215 or 220 in FIG. 8.

In addition, arrows or other indicia of relevancy, such as the arrowextending from block 215 to circle 205 in FIG. 8, can be added toindicate areas of the display to which the information is relevant.

Additional Considerations

1. If the venue 3 in FIG. 1 is an automotive dealership, block 15 mayprovide technical bulletins to the person 6. “Technical bulletin” is aterm of art for circulars prepared by a manufacturer which are sent todealers, which describe specific problems which have been discovered ina particular motor vehicle, and which generally offer solutions.

Many technical bulletins are publicly available, and can be found on theInternet.

2. As another automotive example, the purchaser may wish to learn ofsafety recalls, or other safety issues, involving the vehicle underconsideration. The U.S. government publishes some information of thistype, as at the Consumer Products Safety Commission. Block 15 canprovide access to this.

3. As another automotive example, the purchaser may wish to learn therelative standing of the vehicle to be purchased, relative to similarvehicles. Such standings are reported by independent testing agencies,such as Consumers Union in the United States, which publishes “ConsumerReports,” and various enthusiast magazines written about cars. Thisstanding information may assist the purchaser in bargaining with a salesrepresentative.

Therefore, in one form of the invention, the device 7 contacts thirdparties, which are unaffiliated with the manager of venue 3, such asthose identified in points 1-3 immediately above, in order to collectadditional information about the item to be purchased. This contact canbe done through the Internet, and can be done at the site of the venue3. As stated above, the device 7 can at this time also collectinformation provided by the manager of the venue 3, either through theInternet, through a local network provided by the venue 3, or through athird medium.

In retrieving this information, the person 6 may be given the option ofrequesting a live conference with an assistant, either by telephone orequivalent, or by requesting that the assistant personally assist thecustomer.

4. In one form of the invention, the person 6 retrieves a map of thestore, using one of the approaches described herein. The currentlocation of the shopper on the map is determined, as by having theshopper visually determine his location, and then insert his location onthe map, as by positioning a cursor. Or the device can ascertain theshopper's location without the shopper's involvement. Apparatus forperforming such location are known in the art.

Once the device is equipped with a map of the store, the device displaysinformation needed for the shopper to navigate the store. For example,if the shopper drags a cursor to a location on the map, the map willshow an enlarged view of the region surrounding the cursor. The devicedetermines the coordinates of that region, retrieves information whichdescribes the products in that region, and displays the information.

The shopper can repeat the process on the enlarged view, and eventuallyisolate a specific item, for which the shopper then requestsinformation.

It is pointed out that this procedure can be accomplished at a homecomputer, over the Internet, if the venue 3 provides the proper data andprogramming at a web site.

In another form of the invention, the device is equipped with a camera,and also apparatus which can determine the compass direction in whichthe camera is pointing. For example, assume that the shopper is locatedwithin a rectangular building having its long axis aligned north-south.Assume that the four walls are painted different colors.

The camera, in general, converts images which it sees into bitmaps. Eachbit is associated with a number, which represents color, intensity, andpossible other data. Thus, the bitmap can indicate the direction whichthe camera is pointing, based on the color of the bitmap.

For example, if the camera is pointing directly at the north wall, thecamera will see an entire field of a single color, and that colorindicates the north wall.

If the camera is pointing northwest, for example, processing circuitrywithin the camera can examine the colors seen, and deduce that thedirection is northwest.

5. In one form of the invention, venue 3 is a retail store. The device 7retrieves a master list of products which are offered for sale at thevenue 3. This master list can be obtained from the venue 3 via theInternet.

The person 6 checks off items on the master list which are to bepurchased. As the person 6 travels through the venue 3, his locationwould be ascertained. The device 7 determines, from the master list andthe location of the device 7, which of the selected items are nearby.The device 7 notifies the person 6 of those nearby items.

In this way, the person 6 can generate a shopping list, and travelthrough the venue 6, unguided. The device 7 will signal the person 6when selected items are nearby.

Since the master list, by supposition, was created by the store, theidentification of the products is unambiguous, from the store'sperspective. That is, each item on the master list is directlycorrelated with an item known to be located within the venue.

In contrast, if the person 6 generated his own list, and included theitem “Brand G chocolate cake mix,” such a cake mix may not be present atthe venue 3. Or the actual trade name of the item may be “Brand GExquisitely Chocolate Cake mix.” This latter name, given by the person,may not be recognized by the venue 3.

The use of the master list assures that correct nomenclature is used,because the items are required to be selected from a universe ofpossible items which is defined by the venue 3.

6. The approach of the master list, described immediately above, can beapplied to museums and travel destinations generally. The museum ordestination provides a master list of items to be visited, and device 7indicates when those items are nearby, during the movements of theperson 6.

7. It is pointed out that, in some embodiments, device 7 may not be theagency which deduces its location within the venue 3. For example,equipment associated with server 9 may determine the location of device7, and then transmit information to device 7 about nearby items. In thiscase, device 7 may not learn of its own position, in terms of thespecific coordinate data computed by server 9.

8. The location of the device 7 within the venue 3 can be deduced basedon image recognition. For example, device 7 in FIG. 5 can use its camera30 in FIG. 1 to capture an image along direction D in FIG. 5. That imagecan be sent to a computer of the venue 3, which would recognize theimage, and deduce the location from which the image was taken.

Alternately, items 53 in FIG. 5 can take the form of markers, whichcontain visible symbols, or other symbols which a computer can recognizefrom a digital photograph taken of the symbols. The device 7 takes animage of a nearby marker 53 and transmits the image to the computer ofthe venue 3. The latter deduces the location of the marker, and thus thelocation of the device 7.

9. In one form of the invention, the person 6 in FIG. 1 browses throughthe venue 3 and takes images of selected items, using camera 30. Eachitem is identified, as described herein, and compiled into a list. Inone form of the invention, the person 6 is not involved in making theidentifications, except passively by taking photographs.

The list can act as a “wish list” for items to be purchased later, ifthe venue 3 is a retail store. The venue 3 can provide, to the device 7,addresses of web sites, that is, URLs, Uniform Resource Locations, whenit provides the identities of the items photographed. The web sites canidentify the manufacturers of the items, or otherwise provide furtherinformation about the items.

The list can act as a research assignment, if the venue 3 is a museum ortravel destination. In the case of a museum, the venue 3 can assist theresearch by providing addresses of web sites when it provides theidentities of the items photographed. The web sites provide furtherinformation about the items.

10. Some of the computation described above can be done cooperatively,between the device 7 and the venue's systems.

Numerous substitutions and modifications can be undertaken withoutdeparting from the true spirit and scope of the invention. What isdesired to be secured by Letters Patent is the invention as defined inthe following claims.

1. A method of operating a portable computing device in cooperation witha venue comprising a retail store which contains a collection of itemsfor sale, comprising: a) loading into the device a shopping list ofitems, in which each listed item identifies unambiguously acorresponding item in the collection of the store; b) traveling throughthe store with the device; and c) receiving an indication from thedevice when the device is adjacent an item listed on the shopping list.2. A method of operating a portable computing device in cooperation witha venue which contains a collection of items, comprising: a) loadinginto the device a list of items, in which each listed item identifiesunambiguously a corresponding item in the collection of the venue; b)traveling through the venue with the device; and c) receiving anindication from the device when the device is adjacent an item listed onthe list.
 3. Method according to claim 2, in which the venue comprises atravel destination, and the items comprise points of interest totourists.
 4. Method according to claim 2, in which the venue comprises amuseum, and the items comprise exhibits within the museum. 5-18.(canceled)